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The  OJflce  of  the  Secretary  of  Defense  chartered  the  Joint  Test  and  Evaluation  Methodology 
project  to  institutionalize  testing  in  a  joint  environment.  The  project  has  now  finished  most  of 
its  major  activities.  In  this  article  we  describe  our  key  accomplishments,  findings,  conclusions, 
and  recommendations.  Testing  in  a  Joint  Environment  refers  to  tests  of  military  systems  as 
participating  elements  in  overarching  joint  systems  of  systems.  The  concept  first  appeared  in 
Strategic  Planning  Guidance  and  was formally  introduced  as  Department  of  Defense  policy  in  a 
roadmap  signed  by  the  Deputy  Secretary  in  2004.  Several  working  groups  were  formed  to 
implement  the  roadmap.  The  Joint  Test  and  Evaluation  Methodology  project  was  in  itiated  in 
2005  to  continue  efforts  of  the  methods  and  processes  working  group.  Throughout  the  past  three 
years  we  have  developed,  tested,  and  evaluated  a  number  of  methods  and  processes  for  defining 
and  using  a  distributed  live,  virtual,  and  constructive  joint  test  environment  to  evaluate  system 
peformance  and joint  mission  effectiveness.  We  briefly  describe  those  processes,  what  we  learned 
by  testing  them,  and  the  extent  to  which  they  improve  the  ability  to  conduct  tests,  across  the 
acquisition  life  cycle,  in  realistic  joint  mission  environments.  We  also  describe  the  results  of  two 
large-scale  distributed  tests — INTEGRAL  FIRE  07  and  Joint  Battlespace  Dynamic 
Deconfliction  08 — which  used  mixes  of  live,  virtual,  and  constructive  elements  to  test  a 
number  of  systems  in  joint  environments.  Several  challenges  remain,  and  we  make 
recommendations  to  continue  progress  toward  the  goals  of  testing  in  a  joint  environment. 

The  Department’s  long-term  strategy  calls  for  evaluations  of  joint  system  effectiveness 
throughout  all  phases  of  all  weapon  systems’  development  and  deployment. 

Key  words:  Acquisition;  joint  test  environment;  joint  mission  effectiveness,  testing, 
methods,  processes,  planning;  planning;  rehearsal-of-concept  (rock)  drills;  simulations, 
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The  Joint  Test  and  Evaluation  Method¬ 
ology  (JTEM)  project  was  initiated  in 
February  2005  by  the  Director  of 
Operational  Test  and  Evaluation 
(DOT&E).  We  were  directed  to  in¬ 
vestigate,  evaluate,  and  make  recommendations  to 
improve  the  ability  to  test  across  the  acquisition  life 
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cycle  in  realistic  joint  mission  environments.  Our  focus 
was  to  be  on  methods  and  processes  for  testing  in  a 
joint  environment.  The  concept  of  “testing  in  a  joint 
environment”  comes  from  U.S.  Department  of  De¬ 
fense  (DoD)  2006-2011  Strategic  Planning  Guidance 
for  Joint  Testing  in  Force  Transformation.  It  refers  to 
tests  of  military  systems  as  participating  elements  in 
overarching  joint  systems-of-systems.  Over  the  past 
three-plus  years,  we  developed  and  applied  several 
processes  and  test  methodologies.  Many  are  refine- 
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merits  to  current  test  and  evaluation  procedures;  but 
some  are  not.  In  this  article  we  discuss  three  of  the 
more  significant  changes  in  test  and  evaluation  (T6cE) 
procedures  needed  to  make  testing  in  a  joint  environ¬ 
ment  a  routine  part  of  defense  system  development. 
First,  testing  in  a  joint  environment  must  be  integrated 
into  each  acquisition  program’s  T&E  strategy.  Second, 
test  events  take  on  several  new  dimensions,  especially 
during  development  testing.  And  third,  the  evaluation 
of  test  results  brings  together  warfighters  and  devel¬ 
opers  in  new  and  challenging  ways. 

Because  the  concept  of  testing  in  joint  environments 
originated  in  transformation  planning  guidance,  it  is 
fundamentally  transformative  in  nature.  And  transfor¬ 
mation,  we  discovered,  is  hard.  The  DoD’s  goal  is  to 
define,  develop,  and  then  test  new  military  systems  in 
the  context  of  how  we  fight,  i.e.,  jointly.  But  while  war 
fighting  is  now  an  inherently  joint  process,  defense 
systems  acquisition  is  inherently  not.  And  that  is  the 
overarching  challenge  to  testing  in  a  joint  environment. 
The  Testing  in  a  Joint  Environment  Roadmap  (DoD, 
2004a),  coordinated  by  DOTScE  in  2004,  remains  the 
only  official  document  directing  testing  in  a  joint 
environment.  The  roadmap  identifies  changes  to 
policy,  procedures,  and  test  infrastructure  needed  to 
routinely  conduct  T&E  in  joint  environments.  The 
approved  roadmap  makes  testing  in  a  joint  environ¬ 
ment  clear  Department  policy  and  calls  for  all 
programs,  regardless  of  acquisition  category,  to  dem¬ 
onstrate  their  joint  capability  early  and  throughout 
their  respective  development  cycles.  But  acquisition 
programs,  by  statute,  are  initiated,  funded,  and 
managed  within  military  services.  The  roadmap  still 
defines  a  desired  end  state,  but  the  Department  has  yet 
to  bridge  the  gap  between  this  end  state  and  current 
practice.  This  theme  is  echoed  in  conclusions  and 
recommendations  discussed  later. 

A  key  aspect  of  JTEM’s  direction  from  DOT&E 
was  using  a  distributed  live,  virtual,  and  constructive 
(LVC)  joint  test  environment  to  evaluate  system 
performance  and  joint  mission  effectiveness.  The 
Testing  in  a  Joint  Environment  Roadmap  authors 
quickly  concluded  that  no  single  test  facility  could 
consistently  provide  sufficiently  robust  joint  environ¬ 
ments.  The  authors  saw  modern  networks  and  rapidly 
improving  simulations  as  the  means  to  overcome 
single-facility  limitations.  Networks  could  make  several 
different  and  geographically  dispersed  test  facilities 
appear  as  one.  Networks  also  allow  operator  or 
hardware-in-the-loop  simulations  (sometimes  called 
virtual  simulations)  to  substitute  for  live  systems  and 
digital  computer  simulations  (sometimes  called  con¬ 
structive  simulations)  to  substitute  for  live  or  virtual 
systems  in  a  joint  test  environment.  Combinations  of 


live,  virtual,  and  constructive  simulations — linked 
through  networks  into  a  single  distributed  environ¬ 
ment — could  then  form  LVC  joint  mission  environ¬ 
ments  for  testing.  A  substantial  portion  of  JTEM 
resources  went  to  systems  engineering  activities  used  to 
integrate  simulations  into  a  distributed  environment. 
As  it  turned  out,  these  technical  activities  were 
relatively  easy  compared  to  the  nontechnical  challenges 
discussed  in  this  article. 

During  the  past  three  years  we  used  various  activities 
as  settings  for  testing  and  evaluating  evolving  versions 
of  methods  and  processes.  Some  observers  have  likened 
this  to  making  a  movie  about  people  who  are  putting 
on  a  play.  Just  as  the  play  is  a  backdrop  for  the  movie 
characters,  JTEM  activities  were  backdrops  to  evaluate 
processes  for  testing  in  a  joint  environment.  Initially 
we  used  Rehearsal  of  Concept  (ROC,  sometimes 
spelled  rocE)  drills  (U.S.  Marine  Corp,  2001)  for  initial 
process  evaluations.  Rock  drills  involved  representative 
users  conceptually  walking  through  processes  without 
actually  conducting  a  test.  We  used  these  drills  for 
initial  process  shakedowns  to  uncover  major  problems 
before  applying  the  processes  during  distributed  LVC 
events.  In  2007  the  distributed  event  was  the  Air 
Force’s  INTEGRAL  FIRE,  and  in  2008  it  was  the 
Army’s  Joint  Battlespace  Dynamic  Deconfliction 
QBD2).  Potential  users  of  JTEM  processes  applied 
selected  processes  during  the  planning  and  conduct  of 
these  events.  Each  event  included  live,  virtual,  and 
constructive  representations  of  systems  that  together 
accomplished  one  or  more  joint  missions.  JTEM 
selected  some  of  these  systems  and  joint  missions  as 
notional  test  items.  We  then  used  data  collected  during 
the  events  to  evaluate  system  performance  and  mission 
effectiveness  in  a  joint  environment. 

The  emphasis  in  this  article  is  on  the  more  difficult 
future  challenges  facing  the  DoD  if  testing  in  a  joint 
environment  is  to  become  an  achievable  goal.  In  terms 
of  current  processes,  these  challenges  start  with  Test 
and  Evaluation  Master  Plans  (TEMPs).  The  next 
section  explains  the  planning  information  needed  in 
TEMPs  to  make  testing  in  a  joint  environment  an 
integral  part  of  an  acquisition  program’s  T&E  strategy. 
Then  we  describe  some  enduring  relationships  among 
test  organizations  needed  to  make  distributed  LVC 
testing  a  routine  part  of  development  and  operational 
tests.  Next  we  identify  how  results  of  tests  in  joint 
environments  must  be  evaluated  concurrently  by 
developers,  operational  testers,  and  end  users.  We 
conclude  with  some  recommendations  that  span  these 
three  areas.  None  of  these  changes  is  prohibited  by 
current  policy,  directives,  or  law.  However,  they 
represent  transformative,  cultural  changes,  and  they 
may  require  substantial  commitments  of  resources. 
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T&E  master  plans 

From  the  beginning  we  recognized  the  importance 
of  addressing  testing  in  a  joint  environment  in  each 
program’s  TEMP.  For  one  thing,  if  testing  in  a  joint 
environment  is  not  part  of  the  TEMP,  then  testing  in  a 
joint  environment  is  not  resourced.  But  while  inte¬ 
grating  testing  in  a  joint  environment  into  master  plans 
seemed  straightforward  at  first,  it  turned  out  to  more 
difficult  than  we  expected.  We  gathered  information 
on  TEMP  modifications  for  testing  in  a  joint 
environment  during  workshops  with  the  Operational 
Test  Agencies  (OTAs)  and  as  part  of  early  planning  for 
one  of  our  distributed  test  events.  The  OTA  workshop 
concentrated  on  those  parts  of  a  TEMP  of  most 
interest  to  an  OTA — Initial  Operational  Test  and 
Evaluation  (IOT8cE)  to  support  a  full  rate  production 
decision,  operational  assessments  conducted  periodi¬ 
cally  throughout  development  testing,  and  resourcing 
for  both.  Early  planning  for  our  second  distributed  test 
event  attempted  to  broadly  define  the  joint  operational 


context  for  the  tests  using  guidance  from  current  joint 
doctrine  and  tasks. 

During  our  workshop,  the  OTAs  had  many 
questions  about  how  to  build  a  TEMP  incorporating 
testing  in  a  joint  environment,  but  two  are  particularly 
noteworthy.  Senior  technical  leaders  were  looking  for 
guidance  on  how  to  insert  testing-in-a-joint-  environ¬ 
ment  events  into  the  overall  test  schedule.  One  sensible 
answer  is  to  have  OTAs  conduct  testing  in  a  joint 
environment  during  traditional  Operational  Assess¬ 
ments  (OAs)  or  even  early  operational  assessments. 
OAs  do  not  carry  the  same  restrictions  on  simulation 
use  as  IOT8cE.  Hence  OTAs  could  provide  valuable 
operational  insight  into  design  alternatives  when  the 
developer  may  be  working  with  relatively  easy-to- 
change  constructive  or  virtual  prototypes.  And  if  such 
events  are  to  be  conducted  in  realistic  joint  mission 
environments,  then  OTAs  are  better  positioned  to  plan 
them.  This  leads  to  the  second  question:  Because  joint 
mission  environments  necessarily  include  tactics,  tech- 
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and  missions  to  meet 
customer  objectives 

•  Integrate  multiple  test 
requirements 


•  Merge  single-customer 
requirements  into  single 
execution  sequence 


environment  using 
distributed  LVC  assets 

•  Ensure  network 
connectivity  and 
approvals  to  operate 


•  Identify  data  collection 
requirements  from  customer 
test  plans 

•  Define  test  instrumentation 


•  Integrate  distributed 
components  so  each 
communicate  and 
interact  properly 

Figure  2.  Most  effective  functionai  organization  based  on  JTEM  events. 


•  Process  data  into  forms  to 
meet  customer  requirements 

•  Manage  security 


niques,  and  procedures  (TTPs)  for  employing  the 
system  under  test,  how  do  you  plan  for  situations  when 
current  TTPs  are  clearly  inappropriate  for  the  new 
system?  A  complicating  factor,  as  we  discuss  later,  is 
that  system  effectiveness  may  depend  on  TTPs,  and 
vice  versa. 

We  also  addressed  some  TEMP  information  during 
early  planning  for  our  second  distributed  event.  The 
Army  used  the  JBD2  event  to  integrate  distributed 
components  in  support  of  future  test  requirements. 
Working  with  Army  event  coordinators,  our  intent  was 
to  define  a  broad  joint  operational  context  for  the  test 
based  on  current  joint  doctrine  (DoD,  2004b)  and  the 
Universal  Joint  Task  List.  However,  as  might  be 
expected,  we  found  current  doctrine  and  training  tasks 
a  poor  fit  for  future  capabilities.  We  were  also 
hampered  by  the  lack  of  documentation  from  the  Joint 
Capabilities  Integration  and  Development  System 
(JCIDS)  (DoD,  2007).  Capability  Development  Doc¬ 
uments  and  Capability  Production  Documents,  for 
example,  should  address  future  doctrine  adjustments 
that  will  be  needed  when  the  capability  is  fielded.  And 
certainly  these  doctrine  requirements  should  be  used  as 
a  starting  point  for  testing-in-a-joint-environment 
operational  concepts.  When  available  and  appropriate. 


another  good  starting  point  would  be  joint  architec¬ 
tures  provided  by  U.S.  Joint  Forces  Command 
QFCOM).  Figure  1  shows  an  example  for  close  air 
support  (DoD,  2003). 

But  normally  the  effectiveness  of  proposed  future 
doctrine  will  be  uncertain  until  after  some  field  trials. 
We  have  concluded  that  a  sensible  approach  to  master 
plans  for  testing  in  a  joint  environment  is  to  test 
nonmaterial  doctrinal  concepts  along  with  the  material 
solution  to  a  joint  capability  gap.  But  this  is  not  how 
most  acquisition  programs  are  currently  managed. 

Relationships  among  test  organizations 

An  important  objective  of  JTEM  distributed  events 
was  to  evaluate  the  effectiveness  and  suitability  of 
working  group  structures.  Our  goal  was  to  use  these 
evaluations  to  recommend  organizational  relationships 
and  functions  appropriate  for  a  persistent  distributed 
LVC  test  range.  This  range  should  be  able  to  support 
future  testing  in  a  joint  environment  on  a  regular  basis. 
INTEGRAL  FIRE  used  three  primary  working 
groups  and  one  overarching  group  to  coordinate 
among  the  first  three.  JBD2  created  six  primary 
working  groups.  Combining  lessons  learned  from  these 
two  constructs,  we  identified  three  basic  functional 
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areas  needed  to  effectively  conduct  distributed  LVC 
testing.  Within  a  single  test  organization,  these 
activities  might  fall  within  current  range  management, 
range  operations,  and  data  management  units.  For  our 
distributed  events,  involving  multiple  test  organiza¬ 
tions,  the  functional  areas  were  operations,  LVC 
integration,  and  data-related  functions.  Figure  2  in¬ 
cludes  a  few  more  details.  Two  fundamental  assump¬ 
tions  behind  Figure  2  are  (a)  multiple  test  customers 
are  participating  in  each  test  event,  and  (b)  each 
individual  customer  has  an  approved  test  plan.  The 
latter  mirrors  typical  single-range  procedures  where  a 
customer  cannot  enter  the  scheduling  process  without 
an  approved  test  plan  or  similar  document.  Test 
organizations  should  consider  some  permanent  cross- 
organizational  relationships  to  accomplish  the  func¬ 
tions  in  Figure  2,  including  approval  procedures  for 
test  plans  requiring  distributed  LVC  events. 

A  few  other  aspects  of  Figure  2  require  some 
clarification.  For  example,  JCIDS  adjudication  would 
entail  resolving  real  or  apparent  inconsistencies  among 
joint  mission  requirements.  Distributed  configuration 
management  is  clearly  necessary  and  might  best  be 
handled  by  a  group  encompassing  configuration 
managers  at  participating  test  organizations.  We 
should  also  point  out  INTEGRAL  FIRE  and  JBD2 
had  compressed  timelines  and  focused  on  single  events 
at  predetermined,  fixed  dates.  Activities  were  neces¬ 
sarily  focused  on  constructing  distributed  environ¬ 
ments,  executing  operational  missions,  and  collecting 
data  with  whatever  assets  happened  to  be  ready  on  test 
day.  Early  test  planning  was  rushed,  and  detailed  test 
planning  was  inconsistent  across  events  and  customers. 
INTEGRAL  EIRE  had  fewer  problems,  due  in  no 
small  part  to  its  overarching  coordinating  group. 
Hence  such  a  coordinating  function  will  be  critical  to 
the  success  of  future  distributed  LVC  tests. 

Evaluation  of  test  results 

We  defined  notional  test  items — systems  and 
associated  joint  missions — during  our  distributed 
LVC  events  to  create  opportunities  to  apply  our 
proposed  processes  for  the  evaluation  of  test  results. 
For  example,  INTEGRAL  FIRE  included  a  construc¬ 
tive  network  enabled  weapon  (system  under  test) 
employed  in  support  of  joint  close  air  support  missions. 
INTEGRAL  EIRE  also  included  a  constructive 
surface-to-surface  missile  used  for  joint  fire  support 
missions  (DoD,  2006).  During  test  trials,  calls  for  fire 
support  and  air  support  requests  were  sequenced  to 
intentionally  create  airspace  conflicts.  Conflicts  were 
then  resolved  using  current  joint  airspace  control 
doctrine  (DoD,  2004).  The  constructive  network 
enabled  weapon  (NEW)  was  an  air-launched,  bomb- 


Figure  3.  Procedures  used  in  INTEGRAL  FIRE  to  handoff 
weapon  control. 


on-coordinates,  sub-500-pound-class  guided  bomb 
with  data  link  capabilities.  The  weapon’s  data  link 
mode  with  third  party  targeting  was  evaluated  in  these 
tests.  In  this  mode,  target  coordinates  stored  in  the 
weapon  are  updated  after  launch  by  either  another 
aircraft  or  a  ground-based  Joint  Terminal  Attack 
Controller.  Before  the  weapon  will  accept  updated 
coordinates,  the  launching  aircraft  must  hand  off 
control  of  the  weapon  to  one  of  these  third  parties. 
The  constructive  NEW  model  was  designed  to 
implement  handoff  procedures  contained  in  draft 
operating  concepts  (Air  Combat  Command,  2006). 
These  procedures  are  outlined  in  Figure  3. 

Eor  the  purposes  of  providing  an  example  applica¬ 
tion  of  JTEM  evaluation  processes,  postlaunch  hand¬ 
overs  of  NEW  control  were  conducted  to  determine 
the  ability  of  pilots  and  Joint  Terminal  Attack 
Controllers  to  perform  handoff  functions  in  a  joint 
mission  environment.  Each  trial  included  a  postlaunch 
handoff  of  NEW  control  by  the  launching  aircraft. 
Prior  to  weapon  launch,  pilots  of  the  launching  aircraft 
coordinated  handover  of  weapon  control  to  either  a 
Joint  Terminal  Attack  Controller  or  a  second  aircraft. 
Handoff  time  (time  interval  from  Tj- to  T^)  measured 
the  effectiveness  of  each  handoff.  Test  results  showed 
that  handoffs  to  the  second  aircraft  were  relatively  fast 
when  the  airspace  control  volume  was  small,  but 
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relatively  slow  when  larger  airspace  control  volumes 
were  used.  This  result  indicates  interdependence 
between  joint  airspace  control  doctrine  and  weapon 
design  mechanization  during  the  test.  The  system- 
acquisition  question  is:  Should  the  developer  modify 
the  weapon  design  or  should  the  operational  commu¬ 
nity  modify  doctrine?  Who  decides?  We  recommend 
the  DoD  clarify  responsibilities  to  account  for  these 
inevitable  material-nonmaterial  dependencies.  We  also 
believe  better  guidance  is  needed,  in  general,  on  how 
evaluations  of  joint  mission  effectiveness  are  to  be  used 
by  milestone  decision  authorities  to  support  decisions 
such  as  continued  development,  fuU-rate  production, 
or  fielding. 

Summary 

Through  our  rock  drills,  distributed  LVC  events, 
and  related  activities,  the  JTEM  project  has  been  able 
to  sustain  some  momentum  toward  institutionalizing 
testing  in  a  joint  environment.  In  addition  to  the 
conclusions  and  recommendations  discussed,  our  final 
report  will  contain  many  other  technical  and  nontech¬ 
nical  findings.  For  example,  test  infrastructure  invest¬ 
ments  are  currently  not  managed  with  a  distributed, 
joint  test  environment  in  mind.  And  the  opposing- 
force  side  of  the  equation  remains  largely  ad  hoc 
(although  the  Test  and  Evaluation  Threat  Resource 
Activity  has  jumped  in  to  tackle  some  aspects  of  the 
problem).  Overall,  JTEM  has  contributed  to  building 
the  foundation  for  a  solid  community  of  interest  for 
distributed  LVC  testing.  Test  organizations  across  the 
services  are  now  better  prepared  to  support  testing  in  a 
joint  environment  when  requirements  are  formally 
communicated  to  acquisition  program  managers.  □ 
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